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[ DOCUMENT NAME ] 



Specification 



[ TITLE OF THE INVENTION ] 



Apparatus for synchronized playback of audio 



data and performance data and method therefor 
[ SCOPE OF THE PATENT CLAIM ] 
[ Claim 1 ] 

A recorder being characterized by having: 

a first receiving means for receiving audio data representing audio waveform of a 
musical tune; 

a second receiving means for receiving control data instructing performance 

control; 

a first generating means for generating a first index indicating a variation 
tendency for a predetermined time period of the aforesaid audio data; 

a second generating means for generating a second index indicating a variation 
tendency for a time period longer than the aforesaid predetermined time period of the 
aforesaid audio data; and 

a recording means for recording performance data comprising the aforesaid 
control data and a first time data representing a receiving timing of the control data, 
comparing the aforesaid first index and the aforesaid second index, detecting rapid 
variation of the aforesaid audio data and recording a second time data representing a timing 
pf the aforesaid rapid variation. 

[ Claim 2 ] 

A player being characterized by having: 

a first receiving means for receiving performance data having control data for 
instructing performance control and a first time data instructing an execution timing of said 
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performance control and second time data representing a timing of a rapid variation of the 
audio waveform; 

a second receiving means for receiving audio data representing an audio . 
waveform of a musical tune; 

a first generating means for generating a first index indicating a variation 
tendency for a predetermined time period of the aforesaid audio data; 

a second generating means for generating a second index indicating a variation 
tendency for a time period longer than the aforesaid predetermined time period of the 
aforesaid audio data; 

a third generating means for comparing the aforesaid first index and the aforesaid 
second index in order to detect a rapid variation of the aforesaid audio data and generating 
a third time data representing a timing of the aforesaid rapid variation; 

an adjusting means for adjusting the aforesaid first time data based on the time 
information represented by the aforesaid second time data and the aforesaid third time 
data; and 

a transmission means for transmitting the aforesaid control data at the timing in 
compliance with the aforesaid first time data adjusted by the aforesaid adjusting means. 
[ Claim 3 ] 

A recording method being characterized by having: 

a first receiving step for receiving audio data representing audio waveform of a 
musical tune; 

a second receiving step for receiving control data instructing performance control; 
a first generating step for generating a first index indicating a variation tendency 
for a predetermined time period of the aforesaid audio data; 
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a second generating step for generating a second index indicating a variation 
tendency for a time period longer than the aforesaid predetermined time period of the 
aforesaid audio data; and 

a recording step for recording performance data comprising the aforesaid control 
data and a first time data representing a receiving timing of the control data, comparing the 
aforesaid first index and the aforesaid second index, detecting rapid variation of the 
aforesaid audio data and recording a second time data representing a timing of the 
aforesaid rapid variation. 

[ Claim 4 ] 

A playback method being characterized by having: 

a first receiving step for receiving performance data having control data for 
instructing performance control and a first time data instructing an execution timing of said 
performance control and second time data representing a timing of a rapid variation of the 
audio waveform; 

a second receiving step for receiving audio data representing an audio waveform 
of a musical tune; 

a first generating step for generating a first index indicating a variation tendency 
for a predetermined time period of the aforesaid audio data; 

a second generating step for generating a second index indicating a variation 
tendency for a time period longer than the aforesaid predetermined time period of the 
aforesaid audio data; 

a third generating step for comparing the aforesaid first index and the aforesaid 
second index in order to detect a rapid variation of the aforesaid audio data and generating 
a third time data representing a timing of the aforesaid rapid variation; 



3 



Submission Date : the 14th year of Heisei, August 22 
Ref. No. - C 30594 Page 

an adjusting step for adjusting the aforesaid first time data based on the time 
information represented by the aforesaid second time data and the aforesaid third time 
data; and 

a transmission step for transmitting the aforesaid control data at the timing in 
compliance with the aforesaid first time data adjusted by the aforesaid adjusting means. 
[ Claim 5 ] 

A program which causes a computer to execute: 

a first receiving step for receiving audio data representing audio waveform of a 
musical tune; 

a second receiving step for receiving control data instructing performance control; 

a first generating step for generating a first index indicating a variation tendency 
for a predetermined time period of the aforesaid audio data; 

a second generating step for generating a second index indicating a variation 
tendency for a time period longer than the aforesaid predetermined time period of the 
aforesaid audio data; and 

a recording step for recording performance data comprising the aforesaid control 
data and a first time data representing a receiving timing of the control data, comparing the 
aforesaid first index and the aforesaid second index, detecting rapid variation of the 
aforesaid audio data and recording a second time data representing a timing of the 
aforesaid rapid variation. 

[ Claim 6 ] 

A program which causes a computer to execute: 

a first receiving step for receiving performance data having control data for 
instructing performance control and a first time data instructing an execution timing of said 
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performance control and second time data representing a timing of a rapid variation of the 
audio waveform; 

a second receiving step for receiving audio data representing an audio waveform 
of a musical tune; 

a first generating step for generating a first index indicating a variation tendency 
for a predetermined time period of the aforesaid audio data; 

a second generating step for generating a second index indicating a variation 
tendency for a time period longer than the aforesaid predetermined time period of the 
aforesaid audio data; 

a third generating step for comparing the aforesaid first index and the aforesaid 
second index in order to detect a rapid variation of the aforesaid audio data and generating 
a third time data representing a timing of the aforesaid rapid variation; 

an adjusting step for adjusting the aforesaid first time data based on the time 
information represented by the aforesaid second time data and the aforesaid third time 
data; and 

a transmission step for transmitting the aforesaid control data at the timing in 
compliance with the aforesaid first time data adjusted by the aforesaid adjusting means. 
[ DETAILED EXPLANATION OF THE INVENTION ] 
[ 0001 ] 

[ TECHNICAL FIELD OF THE INVENTION ] 

The present invention relates to an apparatus and a method for playing back 
performance data including information relating to performance control of a musical tune 
in synchronization with playback of audio data. 

[ 0002 ] 
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[ PRIOR ART ] 

There is an apparatus for reading out audio data from a recording medium such as 
a music CD (Compact Disc) and generating sounds from the readout audio data to be 
output as a means for playing back a musical tune. There is an automatic performance 
apparatus for reading out data including information on performance control of a musical 
tune from a recoding medium such as an FD (Floppy Disk) and for controlling tone 
generation of a tone generator by using the readout data as another means for playing back 
the musical tune. There is MIDI data created by complying with the MIDI (Musical 
Instrument Digital Interface) standard as the data including information relating 
performance control of the musical tune. 

[ 0003 ] 

Recently, a method for synchronizing the automatic performance with MIDI data 
with the playback of audio data recorded on the music CD has been proposed. There is a 
method of using time codes recorded on the music CD as one of them (see patent 
document 1 and patent document 2, for example). The methods will be explained below. 

[ 0004 ] 

First, the audio data and time codes in the music CD are reproduced by a music 
CD player. Then, the audio data is output as a sound and the time codes are supplied to a 
recorder. Herein, the time code is data associated with the certain amount of audio data 
and each time code represents a lapse of time from the a start of the musical tune to a 
playback timing of the audio data associated with said time code. The musical instrument 
is played with the playback of the music CD and the MIDI data are sequentially supplied 
from the musical instrument to the recorder. The recorder receives the MIDI data from 
the musical instrument and records the MIDI data on the recording medium with the time 
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information representing the receiving timing. The recorder receives the time code from 
the music CD player and records this in the recording medium with the time information 
representing receiving timing. As a result, a file in which the time code and the MIDI 
data are mixed is created in the recording medium. In this file, respective time codes and 
the MIDI data have time information representing lapses of time from the musical tune 
playback starting timing until respective playback timing. 
[ 0005 ] 

Thus, the audio data of the same musical tune is played back from a music CD 
after the MIDI data and the time code are recorded on the recording medium, the MIDI 
data is read out from the recording medium in synchronization therewith and the automatic 
performance is realized. The operations are as follows. 

[ 0006 ] 

First, the audio data and the time code are reproduced from the music CD by a 
music CD player. Then, the audio data is output as sound and the time code is supplied to 
the player of the MIDI data. The player reads out the MIDI data stored in the file based 
on the time information recorded therewith and sequentially transmits those to a musical 
instrument capable of automatic performance with the MIDI data. At that time, the player 
adjusts the time difference between the playback of the audio data of the music CD and the 
playback of the MIDI data based on the time code received from the CD player and the 
time code read out from the file together with the MIDI data. As a result, the 
synchronized playback of the audio data of the music CD and the MIDI data are realized. 

[ 0007 ] 

[ Patent Publication 1 ]: Patent Application No. 2002 - 7872 
[ Patent Publication 2 ]: Patent Application No. 2002 - 7873 
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[ 0008 ] 

[ PROBLEM TO BE SOLVED BY THE INVENTION ] 

However, the synchronized playback of the audio data of the music CD and the 
MIDI data is not possible by a method of using the time codes of the music CD for music 
CDs labeled with different time codes for the same musical tune. 

[ 0009 ] 

Currently, there are lots of music CDs of different versions for the same musical 
tune. The music CD usually records audio data which the master data of the musical tune 
is edited by adding acoustic effects such as reverberation thereon. In the music CD of the 
different version, an offset time until the musical tune starts may differ. A clock speed 
used for recording the audio data after editing may not coincide with a clock speed used for 
recording the master data. As a result, the music CD of the different version, which 
records the audio data edited differently, may be labeled with time codes having slightly 
different time periods. In such a case, when the MIDI data for synchronized performance 
created with a conventional technology using the time codes is used for the music CD of 
the different version for the same musical tune, the musical tune from the music CD and 
the performance with the MIDI data are shifted at the start of the musical tune, or the 
musical tune of the music CD and the performance with the MIDI data gradually become 
shifted even though the musical tune of the music CD and the performance with the MIDI 
data are synchronized at the start of the musical tune. 

[0010] 

Accordingly, it encounters a problem that the music CD recording the audio data 
for the same musical tune based on the conventional technology using the time codes needs 
different MIDI data for a synchronized performance depending on a variation of time 
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intervals of the time codes attached to a version of the music CD, respectively. 
[0011] 

By contemplating the above circumstances, it is an object of the present invention 
to provide a recorder, a player, a recording method, a playback method and program of 
performance data such as MIDI data or the like to be played back in synchronization with 
plural versions of audio data recorded at difference time intervals due to different clock 
speeds used upon recording for audio data of the same musical tune. 

[0012] 

[ MEANS TO SOLVE THE PROBLEM ] 
[0012] 

To solve the above explained problem, the present invention provides a recorder 
being characterized by having: 

a first receiving means for receiving audio data representing audio waveform of a 
musical tune; 

a second receiving means for receiving control data instructing performance 

control; 

a first generating means for generating a first index indicating a variation 
tendency for a predetermined time period of the aforesaid audio data; 

a second generating means for generating a second index indicating a variation 
tendency for a time period longer than the aforesaid predetermined time period of the 
aforesaid audio data; and 

a recording means for recording performance data comprising the aforesaid 
control data and a first time data representing a receiving timing of the control data, 
comparing the aforesaid first index and the aforesaid second index, detecting rapid 
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variation of the aforesaid audio data and recording a second time data representing a timing 
of the aforesaid rapid variation. 
[0013] 

The present invention provides a player being characterized by having: 
a first receiving means for receiving performance data having control data for 
instructing performance control and a first time data instructing an execution timing of said 
performance control and second time data representing a timing of a rapid variation of the 
audio waveform; 

a second receiving means for receiving audio data representing an audio 
waveform of a musical tune; 

a first generating means for generating a first index indicating a variation 
tendency for a predetermined time period of the aforesaid audio data; 

a second generating means for generating a second index indicating a variation 
tendency for a time period longer than the aforesaid predetermined time period of the 
aforesaid audio data; 

a third generating means for comparing the aforesaid first index and the aforesaid 
second index in order to detect a rapid variation of the aforesaid audio data and generating 
a third time data representing a timing of the aforesaid rapid variation; 

an adjusting means for adjusting the aforesaid first time data based on the time 
information represented by the aforesaid second time data and the aforesaid third time 
data; and 

a transmission means for transmitting the aforesaid control data at the timing in 
compliance with the aforesaid first time data adjusted by the aforesaid adjusting means. 
[0014] 
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The present invention provides a recording method being characterized by having: 
a first receiving step for receiving audio data representing audio waveform of a 
musical tune; 

a second receiving step for receiving control data instructing performance control; 

a first generating step for generating a first index indicating a variation tendency 
for a predetermined time period of the aforesaid audio data; 

a second generating step for generating a second index indicating a variation 
tendency for a time period longer than the aforesaid predetermined time period of the 
aforesaid audio data; and 

a recording step for recording performance data comprising the aforesaid control 
data and a first time data representing a receiving timing of the control data, comparing the 
aforesaid first index and the aforesaid second index, detecting rapid variation of the 
aforesaid audio data and recording a second time data representing a timing of the 
aforesaid rapid variation. 

[0015] 

The present invention provides a playback method being characterized by having: 
a first receiving step for receiving performance data having control data for 
instructing performance control and a first time data instructing an execution timing of said 
performance control and second time data representing a timing of a rapid variation of the 
audio waveform; 

a second receiving step for receiving audio data representing an audio waveform 
of a musical tune; 

a first generating step for generating a first index indicating a variation tendency 
for a predetermined time period of the aforesaid audio data; 
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a second generating step for generating a second index indicating a variation 
tendency for a time period longer than the aforesaid predetermined time period of the 
aforesaid audio data; 

a third generating step for comparing the aforesaid first index and the aforesaid 
second index in order to detect a rapid variation of the aforesaid audio data and generating 
a third time data representing a timing of the aforesaid rapid variation; 

an adjusting step for adjusting the aforesaid first time data based on the time 
information represented by the aforesaid second time data and the aforesaid third time 
data; and 

a transmission step for transmitting the aforesaid control data at the timing in 
compliance with the aforesaid first time data adjusted by the aforesaid adjusting means. 
[0016] 

The present invention provides a program which causes a computer execute the 
processes using these recording method and the playback method. 
[0017] 

The present invention provides a recording medium recording performance data 
having control data for instructing performance control and a first time data instructing an 
execution timing of said performance control and a second time data representing a timing 
of a rapid variation of the audio waveform. 

[0018] 

Using the apparatus, method, program and recording medium with said 
configuration, the time data for the time adjustment is created at a timing when the audio 
waveform of the musical tune rapidly changes upon recording and playback of the 
performance data and the transmission timing of the control data by comparing the time 
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data generated upon recording and the time data generated upon playback. As a result, 
the synchronized playback of the audio data and the control data is realized. 
[0019] 

The recorder according to the present invention may include a third receiving 
means for receiving the time codes representing the playback timing of the aforesaid audio 
data and the recording means may generate the aforesaid first time data and the aforesaid 
second time data based on the time information represented by the aforesaid time code. 

The player according to the present invention may include a third receiving means 
for receiving the time codes representing the playback timing of the aforesaid audio data 
and the aforesaid third generating means may generate the aforesaid third time data based 
on the aforesaid time code and the aforesaid transmission means may transmit the 
aforesaid control data based on the time information represented by the aforesaid time code. 

[ 0020 ] 

By using the recorder and the player with this configuration, the time is measured 
in accordance with the time codes with respect to the audio data which is played back by 
the player having a bias in the playback speed so that the control data is synchronously 
played back. 

[0021 ] 

In the recorder according to the present invention, the first generating means 
filters the audio waveform represented by the audio data and extracts the frequency 
component equal to or lower than the first frequency in order to generate the aforesaid first 
index; the aforesaid second generating means filters the audio waveform represented by the 
audio data and extracts the frequency component equal to or lower than the second 
frequency which is lower than the aforesaid first frequency in order to generate the 
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aforesaid second index. 

In the player according to the present invention, the aforesaid first generating 
means filters the audio waveform represented by the aforesaid audio data and extracts the 
frequency components equal to or lower than the first frequency in order to generate the 
aforesaid first index, and the aforesaid second generating means filters the audio waveform 
represented by the aforesaid audio data and extracts the frequency components equal to or 
lower than the second frequency lower than the first frequency in order to generate the 
aforesaid second index. 

[ 0022 ] 

By using the recorder and player with such a configuration, the time data for time 
adjustment having preferable time intervals and the accuracy. 
[ 0023 ] 

In the recorder implemented by the present invention, the aforesaid recording 
means may record the aforesaid second time data in the aforesaid audio data only when the 
rapid variation has not been detected in the predetermined time period from the timing 
when the aforesaid rapid variation is detected. 

In the player implemented by the present invention, the aforesaid third generating 
means may generate the aforesaid third time data only when the rapid variation has not 
been detected in the predetermined time period from the timing when the aforesaid rapid 
variation is detected. 

[ 0024 ] 

By using the recorder and the player according to this configuration, the time 
intervals of the time adjustment is properly maintained even when the audio waveform 
frequently and rapidly varies so that the adjacent time data are not mixed. 
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[0025] 

The player according to the present invention, the aforesaid adjusting means may 
adjust the playback timing of the aforesaid performance data based on valued represented 
by a regression line or a regression curve acquired by the regression analysis on the 
aforesaid second time data and the aforesaid third time data. 

[0026] 

By using the player with such configuration, since the error in the time data for 
the time measurement is adjusted as a whole, the accuracy of the transmission timing of the 
control data in the synchronized playback is enhanced. 

[ 0027 ] 

[ EMBODIMENT OF THE INVENTION ] 
[ 1 ] First embodiment 
[1.1] Structure, function and data format 
[1.1.1] Whole configuration 

Fig. 1 is a view to show a configuration of a synchronized recorder and player SS 
implemented by a first embodiment of the present invention. The synchronized recorder 
and player SS comprises a music CD drive 1, an FD drive 2, an automatic player piano 3, a 
tone generating portion 4, a manipulating display 5, and a controller 6. 

[0028] 

The music CD drive 1, FD drive 2, automatic player piano 3, tone generating 
portion 4 and manipulating display 5 are connected to the controller 6 by communication 
lines, respectively. The automatic player piano 3 and the tone generating portion 4 are 
directly connected each other by the communication line. 

[ 0029 ] 
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[ 1.1.2] Music CD drive 

The audio data stored in the music CD includes audio data representing audio 
information and time codes representing playback timings of the audio data. The music 
CD drive 1 reads out the audio data from the loaded music CD under instructions from the 
controller 6 and sequentially outputs the audio data included in the audio data. The music 
CD drive 1 is connected to a communication interface 65 in the controller 6 by a 
communication line. 

[ 0030 ] 

The audio data, which is output from the music CD drive 1, is 16 bit digital audio 
data in two channels in left and right quantized at a sampling frequency of 44,100 Hz in 16 
bits. The data output from the music CD drive 1 does not include the time code. Since 
the configuration of the music CD drive 1 is similar to a general music CD drive which is 
capable of outputting the digital audio data, the explanation will be omitted. 

[0031 ] 
[ 1.1.3 ]FD drive 

The FD drive 2 records SMF (Standard MIDI File) in the FD or reads out the 
SMF recorded in the FD in order to transmit the readout SMF. The FD drive 2 is 
connected to the communication interface 65 in the controller 6 by the communication line. 
Since the configuration of the FD drive 2 is similar to a general FD drive, the explanation 
will be omitted. 

[ 0032 ] 
[ 1. 1.4 ] MIDI event and SMF 

The SMF is a file including the MIDI event serving as the performance control 
data complied with the MIDI standard and delta time serving as data representing the 



16 



Submission Date : the 14th year of Heisei, August 22 
Ref. No. = C 30594 Page 

execution timing of the respective MIDI events. The MIDI events and the format of the 
SMF will be explained with reference to Fig. 2 and Fig. 3. 
[0033 ] 

A note-on event, a note-off event and a system exclusive event are shown in Fig. 
2 as examples of the MIDI event. The note-on event is a MIDI event which instructs the 
tone generation of a musical tone and comprises 9nH (n represents a channel number, H 
represents a hexadecimal number, and which will be the same hereinafter) representing the 
tone generation, a note number representing a pitch, and a velocity representing the 
strength of the tone generation (or a velocity of striking a key). Similarly, the note-off 
event is a MIDI event which instructs the tone extinction of a musical tone and comprises 
8nH representing the tone extinction, a note number representing a pitch, and a velocity 
representing {he strength of the tone extinction (or a velocity of releasing a key). On the 
other hand, the system exclusive event is a MIDI event which transmits, receives or 
records data formatted by a product or software manufacturer's discretion, and comprises 
FOH representing a start of the system exclusive event, a data length, data and F7H 
representing an end of the system exclusive event. In such a way, the MIDI event does 
not include time information and is us6d for tone generation, tone extinction of the musical 
tone and other control in real time. 

[ 0034 ] 

Fig. 3 shows the general overview of the SMF format. The SMF includes a 
header chunk and a track chunk. The header chunk includes control data relating data 
format and time units included in the track chunk. The track chunk includes the MIDI 
event and delta time representing an execution timing of respective MIDI events. 

[ 0035 ] 
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The SMF has expressions for delta time, namely, the one is expressed in time 
called as a clock representing the relative time with respect to a MODI event immediately 
before arid the other is expressed by a combination time of hour, minute, second and frame 
to represent the absolute time from the head of the musical tune. To make the 
explanation easy, the delta time is defined as the absolute time from the base time and is 
expressed in second in the explanation below. 

Moreover, the MIDI data is the comprehensive name for data created by 
complying with the MIDI standard in the present specification. 

[ 0036 ] 

[1.1.5] Automatic player piano 

The automatic player piano 3 is a musical tone generator which outputs an 
acoustic piano tone and an electronically synthesized piano tone in response to a key 
manipulation and a pedal manipulation by the user of the synchronized recorder and player 
SS. The automatic player piano 3 generates a MIDI event in response to the key 
manipulation and the pedal manipulation by the user and transmits the generated MIDI 
event. Further, the automatic player piano 3 receives the MIDI event and automatically 
plays with acoustic piano sounds and electronically synthesized piano tones in response to 
the received MIDI events. 

[ 0037 ] 

The automatic player piano 3 comprises a piano 31, key sensors 32, pedal sensors 
33, a MIDI event control circuit 34, a tone generator 35 and a driving part 36. 
[0038 J 

The key sensor 32 and the pedal sensor 33 are provided on each of the plural keys 
and the plural pedals of the piano 3 1 in order to detect the positions of keys and pedals, 
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respectively. The key sensor 32 and pedal sensor 33 transmit the detected position 
information, identification number corresponding to each of the keys and pedals, 
respectively, and detected time information to the MIDI event control circuit 34. 
[ 0039 ] 

The MIDI event control circuit 34 receives the position information of the keys 
and pedals, respectively, from the key sensors 32 and the pedal sensors 33 together with 
the identification information of the keys and pedals and the time information, and 
immediately generates MIDI events such as a note-on event, note-off event or the like from 
the information in order to output the generated MIDI event to the controller 6 and the tone 
generator 35. The MIDI event control circuit 34 receives the MIDI event from the 
controller 6 and transfers the received MIDI event to the tone generator 35 or the driving 
part 36. Further, the MIDI event control circuit 34 is under the instruction of the 
controller 6 to determine which the MIDI event received from the controller 6 is 
transferred to the tone generator 35 or the driving part 36. 

[ 0040 ] 

The tone generator 35 receives the MIDI events from the MIDI event control 
circuit 34 and outputs the sound information of various musical instruments as digital 
audio data in the left and right channels based on the received MIDI events. The tone 
generator 35 electronically synthesizes the digital audio data at a pitch designated by the 
received MIDI event and transmits it to a mixer 41 in the tone generating portion 4. 

[0041] 

The driving parts 36 are provided on the respective keys and pedals of the piano 
3 1 and comprises a group of solenoids for driving these and a control circuit for controlling 
the group of solenoids. When the control circuit of the driving parts 36 receives the MIDI 
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events from the MIDI event control circuit, it adjusts current to be supplied to the solenoid 
provided on a corresponding key or pedal in order to adjust magnetic flux generated by the 
solenoid and the key or pedal is operated in response to the MIDI event. 

[ 0042 ] 
[ 1.1.6 ] Tone generator 

The tone generating portion 4 receives the audio data from the automatic player 
piano 3 and the controller 6 and converts the received audio data into sounds to be output. 
The tone generating portion 4 comprises the mixer 41, a D/A converter 42, an amplifier 43 
and a speaker 44. 

[ 0043 ] 

The mixer 41 is a digital stereo mixer which receives plural sets of digital audio 
data in the two channels, left and right, and converts these into a pair of left and right 
digital audio data. The mixer 4 1 receives the digital audio data from the tone generator 
35 of the automatic player piano 3 and at the same time, receives the digital audio data, 
which is read out by the music CD drive 1 from the music CD, through the controller 6. 
The mixer 41 calculates an average of the received digital audio data and transmits this to 
the D/A converter 42 as a pair of digital audio data in right and left. 

[ 0044 ] 

The D/A converter 42 receives the digital audio data from the mixer 41 and 
converts the received digital audio data into the analog audio signal to be output to the 
amplifier 43. The amplifier 43 amplifies the analog audio signal, which is input from the 
D/A converter 42, and outputs it to the speaker 44. The speaker 44 converts the analog 
audio signal, which is input from and amplified by the amplifier 43, into sounds. As a 
result, the audio data recorded in the music CD and the audio data generated by the tone 
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generator 35 are output from the tone generating portion 4 as stereo sounds. 

[ 0045 ] 
[ 1.1.7 ] Manipulation display 

The manipulation display 5 is a user interface when a user of the synchronized 
recorder and player SS manipulates the synchronized recorder and player SS. 

The manipulation display 5 includes key pads when the user depresses to give 
instructions to the synchronized recorder and player SS and a liquid crystal display for 
confirming the state of the synchronized recorder and player SS. When the key pad is 
depressed by the user, the manipulation display 5 outputs a signal corresponding to the 
depressed key pad to the controller 6. The manipulation display 5 receives bit map data 
including information of characters and figures and displays the characters and figures 
based on the received bit map data on the liquid crystal display. 

[ 0046 ] 
[ 1.1.8] Controller 

The controller 6 controls the entire synchronized recorder and player SS. The 
controller 6 comprises a ROM (Read Only Memory) 61, a CPU (Central Processing Unit) 
62, a DSP (Digital Signal Processor) 63, a RAM (Random Access Memory) 64 and a 
communication interface 65. The components are mutually connected each other through 
a bus. 

[ 0047 ] 

The ROM 61 is a non-volatile memory for storing various kinds of control 
program. The control program, whidh is stored in the ROM 61, includes program for 
general control routine and program which causes the CPU 62 to execute routines for 
recording operations and playback operations of the SMF which will be mentioned later. 
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The CPU 62 is a microprocessor, which executes general purpose processing, and reads 
out the control program from the ROM 61 and executes the control routines in accordance 
with the readout control program. The DSP 63 is a processor which permits processing 
of the digital audio data at a high speed, executes a filtering process necessary for a 
managing event generation process, which will be mentioned later, on the digital audio 
data received from the music CD drive 1 and the FD drive 2 by the processor 6, and 
transmits the acquired data to the CPU 62. The RAM 64 is a volatile memory and 
temporarily stores the data used by the CPU 62 and DSP 63. The communication 
interface 65 is an interface which is capable of transmitting and receiving the digital data in 
various formats, converts the format necessary for digital data transmitted or received 
between the music CD drive 1, FD drive 2, automatic player piano 3, tone generating 
portion 4, and manipulation display 5, and relays the data between the respective devices 
and the controller 6. 

[ 0048 ] 
[ 1.2 ] Operation 

Next, operations of the synchronized recorder and player SS will be explained. 
[ 1.2.1 ] Recording operation 

First, operations of the synchronized recorder and player SS, when a user of the 
synchronized recorder and player SS plays a piano in synchronization with the playback of 
a commercially available music CD and the information of the performance is recorded on 
an FD as the MIDI data, will be explained. The music CD used during the recording 
operation, which will be explained below, is called as a music CD-A in order to 
discriminate the music CD used during the playback operation mentioned later. 

[ 0049 ] 
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[ 1.2. 1. 1 ] Start operation of recording 

The user sets the music CD-A in the music CD drive 1 arid an empty FD in the FD 
drive 2. Subsequently, the user depresses the key pads of the manipulation display 5 
corresponding to the recording start of the performance data. The manipulation display 5 
outputs the signal corresponding to the depressed key pad to the controller 6. 

[0050] 

The CPU 62 of the controller 6 receives the signal corresponding to the recording 
start of the performance data from the manipulation display 5 and transmits a playback 
instruction of the music CD to the music CD drive 1 . In response to the playback 
instruction, the music CD drive 1 sequentially transmits the audio data recorded in the 
music CD-A to the controller 6. The controller 6 receives the data for a pair of right and 
left channels for every 1/44100 second from the music CD drive 1. Hereunder, the data 
values for a pair of the right and left channels are expressed as (R(n), L(n)), and the pair of 
data values or respective sets of data values generated from the pair of data values in the 
data generation routine for correlation discrimination are called as "sample values". The 
R(n) and L(n) represent data values in the right channel and the left channel, respectively, 
and they are either of integers ranging from - 32768 to 32767. n is an integer representing 
an order of the audio data and increases from the start of the data such as 0, 1, 2 .... 

[0051 ] 

[1.2.1.2] Transmission of audio data to tone generating portion 

The CPU 62 receives the sample values from the music CD drive 1 and starts time 
measurement by setting the timing as 0 second. At the same time, the CPU 62 
sequentially transmits the received sample values to the tone generating portion 4. The 
tone generating portion 4 receives the sample values from the controller 6 and sequentially 
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converts it into sounds to be output. As a result, the user can listen to the musical tune 
recorded in the music CD-A. 
[ 0052 ] 

[ 1.2.1.3 ] Generation of management event 

The CPU 62 transmits the received sample values to the tone generating portion 4 
and, at the same time, sequentially stores the received sample values in the RAM 64. The 
CPU 62 stores a predetermined number of sample values in the RAM 64 and transmits an 
execution instruction, which executes a management event generating process explained 
hereunder, to the DSP 63. The number of sample values, which are stored in the RAM 64 
until the CPU 62 transmits the execution instruction of the management event generation 
process to the DSP 63, depends on the filter characteristics used in the process which will 
be explained below. The longer the delay time of the delaying element used in the filter, 
the larger the number becomes. In the following explanation, it is assumed that the CPU 
62 transmits the execution instruction of the management event generating process when 
the sample values for a second, namely, 44,100 units of sample values are stored in the 
RAM 64. 

[ 0053 ] 

The management event generation process is a process including a filter process 
extracting frequency components equal to or lower than a certain frequency on the audio 
waveform represented by the audio data stored in the RAM 64 and a filter process 
extracting frequency components equal to or lower than a frequency which is lower than 
the frequency, and compares the values acquired by the two filter processes in order to 
generate the management event. The management event is a flag to generate time 
information used in a timing adjustment process mentioned later. 
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The management event generating process will be explained with reference to Fig. 

4. 

[ 0054 ] 

The DSP 63 receives an execution instruction of the management event 
generation process from the CPU 62 and reads out first 44100 sets of sample values stored 
in the RAM 64, namely, (R(0), L(0)) ~ (R(44099), L(44099)) (step S 1 ). Hereunder, a 
series of 44100 sample values are called as "raw audio data" and the raw audio data having 
(R(n), L(n)) as the last one is represented as "raw audio data (n)" if the raw audio data has 
to be discriminated. In other words, the raw audio data read out by the DSP 63 first in the 
step SI is a raw audio data (44099). 

[ 0055 ] 

Subsequently, the DSP 63 executes an arithmetic average operation of the left and 
right values of the respective sample values of the raw reference audio data and converts 
the stereo data into a monaural data (step S2). The conversion process into the monaural 
is a process to reduce the workload on the DSP 63 in processes after this step. 

[ 0056 ] 

Subsequently, the DSP 63 calculates absolute values for the series of sample 
values converted into the monaural signal (step S3). The process in the step calculates 
substitute values of power of the respective samples. Since the absolute values have 
smaller values than square values representing the power and are easily processed, the 
present embodiment uses the absolute values instead of the square values of respective 
sample values. Accordingly, the square values may be calculated instead of the absolute 
values of the respective sample values in this step when the performance of the DSP 63 is 
high. 



25 



Submission Date : the 14th year of Heisei, August 22 
Ref. No. = C 30594 Page 

[ 0057 ] 

Subsequently, the DSP 63 filters the series of sample values converted into the 
absolute values in the step S3 through the low pass filter (step S4). Hereinafter, the 
values acquired as a result of the processes in the steps SI to S4 are called as "middle term 
index" and the middle term index corresponding to (R(n), L(n)) is called as a "middle term 
index (n)". Herein, a cut-off frequency of the low pass filter used in the step S4 is set at 
100 Hz. The middle term index (n) indicates a tendency in a middle term in a variation of 
the audio waveform represented by the series of sample values with respect to n. In other 
words, the audio waveform represented by the series of sample values goes up and down in 
a short term and the variations in values of the series of sample values filtered by the low 
pass filter are suppressed by the plural precedent sample values. As a result, the short 
term variation component is removed from the waveform represented by the series of the 
middle term index and the variation in the middle term component and the long term 
component remains. The series of the middle term indices including the middle term 
index (n) as the last one, namely, ... the middle term index (n-2), the middle term index (n- 
1) and the middle term index (n) are created by the process in the step S4. The DSP 63 
stores these middle term indices in the RAM 64 (step S5). 

[ 0058 ] 

Subsequently, the DSP 63 filters the series of the middle term indices acquired in 
the step S4 through a comb filter (step S6). The process in this step is a process which 
extracts the low frequency component equal to or lower than a certain frequency from the 
audio waveform represented by the series of the middle term indices. This step is similar 
to the process using the low pass filter having the cut-off frequency lower than the cut-off 
frequency of the low pass filter used in the step S4, and the comb filter is used instead of 
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the low pass filter in the present embodiment because the comb filter applies lower load on 
the DSP 63 than the low pass filter. 
[ 0059 ] 

Fig. 5 shows a configuration of an example of the comb filter to be employed in 
the step S6. In Fig. 5, a process represented by a square rectangular means a delay 
process and k in z~ k means that the delay time in the delay process is (sampling cycle X k). 
As mentioned previously, the sampling frequency of the music CD is 44100 Hz and the 
sampling period is 1 / 44100 second. On the other hand, the process represented by a 
triangle means a multiplication and a value indicated in the triangle means a coefficient of 
the multiplication. Hereinafter, it is assumed that k = 22050, and the frequency 
components higher than I Hz are almost removed therefrom by the filtering process in the 
step S6. In other words, the audio waveform represented by the series of values acquired 
by the process in the step S6 is acquired by extracting the long term variation component 
by removing the middle term variation component from audio waveform represented by 
the series of the middle term indices. 

[ 0060 ] 

Subsequently, the DSP 63 multiplies the series of values acquired by the filtering 
process in the step S6 by a positive constant h, respectively (step S7). The multiplication 
process by the constant h is a process to adjust the frequency to acquire the positive result 
in a comparing process in a next step S8 and time intervals which acquires the positive 
results in the comparing process generally become shorter when a value of the constant h 
becomes small. If the time interval becomes too long, time intervals of the generated 
management events in step SI 1 below becomes longer and the accuracy of the timing 
adjusting process mentioned later become lower. On the other hand, if the time intervals 
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which the positive results are acquired in the comparing process become too short, the 
positive results are cancelled by a process in step S10 below one after another; the time 
intervals of the generated management events also become longer; and as a result, the 
accuracy of the timing adjusting process becomes lower. Accordingly, the value to 
generate the management events at a proper frequency is empirically used as the value of 
the constant h. 

[0061 ] 

The value acquired as a result of the process in the step S7 is called as a "long 
term index" and the long term index corresponding to (R(n), L(n)) are called as a "long 
term index(n)". A series of the long term indices (n) having the long term index (n) as the 
last one, namely, ... a long term index (n-2), a long term index (n-1), and a long term index 
(n) are generated by the process of the step S7. The DSP 63 stores these long term 
indices in the RAM 64 (step S8). 

[ 0062 ] 

The DSP 63 reads out the middle term index (n) and the long term index (n) from 
the RAM 64 and compares to judge whether the middle term index (n) is equal to or larger 
than the long term index (n) or not (step S9). The middle term index and the long term 
index which are read out by the DSP 63 in the step S9 are a middle term index (44099) and 
a long term index (44099). This comparison process represents that the audio waveform 
represented by the raw audio data varies with a large amplitude in the middle term at a 
timing corresponding to the sampling value (R(n), L(n)). In other words, when the 
volume of the sound included within a frequency band of 1 Hz to 100 Hz in the sound 
waveform of the musical tune increases rapidly, the value of the middle term index 
exceeds the value of the long term index so that the positive result (hereinafter referred to 
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as "Yes") is acquired in the comparison process in the step S9. 
[ 0063 ] 

If the positive result Yes is acquired in the comparison process in the step S9, the 
DSP 63 stores the time at that time in the RAM 64. At the same time, the DSP 63 reads 
out the time records when Yes was required in the comparison process in the step S9 in the 
past and judges whether or not there is a record that Yes was acquired in a time period r 
in the past (step S10). When the shorter time intervals cause the result of the comparing 
process in the step S9 to be Yes, the judgment process in this step avoids generation of 
management events at shorter time intervals in a next step S 1 1 as similarly. If the 
management events are generated at shorter time intervals, it is difficult to correctly 
correlate the recorded management events and management events generated by newly 
acquired audio data in a timing adjusting process mentioned later. The generation of the 
management events is avoided at time intervals equal to or shorter than the time period z 
by the process in the step S 1 0. The value of t is determined in order to set the proper 
time intervals of generated management events. Further, the judgment result in the step 
S 10 at an initial judgment becomes No because there is no preceding step S9. 

[ 0064 ] 

When the negative result (hereinafter referred to as "No") is acquired in the 
judgment process in the step S10, the DSP 63 transmits a management event, which 
represents that the audio waveform indicating the raw audio data suffices a predetermined 
condition at a timing corresponding to (R(n), L(n)) after the above described series of 
processes, to the CPU 62 (step SI 1). 

[ 0065 ] 

If the comparison result in the step S9 becomes No; the judgment result in the step 
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S10 becomes Yes; and the process in the step SI 1 is finished, the DSP 63 stands by until 
the CPU 62 receives new sample values, namely, (R(n+1), L(n+1)), from the music CD 
drive 1 and stores the sample values in the RAM 64. When the new sample values are 
stored in the RAM 64 (step SI 2), the DSP 63 returns the process to the step SI and 
executes the process starting from the above mentioned step SI on the series of sample 
values which the sample values newly stored in the RAM 64 serve as the last one, namely, 
the raw audio data (n+1). 
[ 0066 ] 

The sequence in the above step SI to step S12 is continued by incrementing the 
number of n by one until the user instructs the recording finish of the performance data. 
The above is the explanation on the management event generating process. Fig. 6 is a 
view to show the management event generation when the management event generation 
process is executed on the real audio data. Moreover, it is assumed that a single stage 
(Infinite Impulse Response) filter is used as the low pass filter in the step S4 in order to 
create the graph. It is assumed that the constant h in the step S7 is 4, and the time period 
t in the step S10 is 0.55 second. 

[0067] 

Though there are timings B and C which the middle term index exceeds the long 
term index immediately after a timing A when a first management event is created in Fig. 6, 
the management event is created at the timing A only. This is because the timings B and 
C do not lapse a predetermined time period, namely, 0.55 second after the timing A and the 
judgment result in step S10 becomes Yes so that the process in the step SI 1 is not initiated. 

[ 0068 ] 

[1.2.1.4] Generation of MIDI event 
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While the management event generation processes is executed by the DSP 63, the 
user starts the performance on the piano 31. In other words, the user listens to a musical 
tune of the music CD-A which is output from the tone generating portion 4 and depresses 
keys and manipulates pedals to accompany the musical tune. 

[ 0069 ] 

The performance information on the piano 3 1 by the user are detected as motions 
of the keys and pedals by the key sensors 32 and the pedal sensors 33 and is converted into 
MIDI events by the MIDI event control circuit 34 to be transmitted to the CPU 62 of the 
controller 6. 

[ 0070 ] 

[ 1 .2. 1 .5 ] Recording event in SMF 

As mentioned the above, the CPU 62 receives the management events from the 
DSP 63 during the playback of the music CD-A and the MIDI events from the MIDI event 
control circuit 34 of the automatic player piano 3. Fig. 7 is an illustrative view showing 
the relationships between the generated management events and the generated MIDI events 
with respect to time. The CPU 62 receives the management events at timings which lapse 
0.13 second, 0.81 second, 1.45 seconds, respectively, after time measurement starts, 
namely, from when a first sample value is received from the music CD drive 1 . And the 
CPU 62 receives the MIDI events at timings which lapse 0.49 second, 1.23 seconds, 2.18 
seconds, respectively, from when the time measurement starts. 

[ 0071 ] 

The CPU 62 receives the management event, generates a system exclusive event 
representing the management event, and stores the generated system exclusive event in the 
RAM 64 after attaching a delta time representing the lapse of time at the receiving timing 
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of the management event. As similarly, the CPU 62 receives the MIDI event and stores 
the received MIDI events in the RAM 64 after attaching a delta time representing the lapse 
of time at the receiving timing of the MIDI event. 
[ 0072 ] 

After finishing the playback of the musical tune of the music CD- A and the user 
stops the performance on the piano 31, the user depresses a key pad of the manipulation 
display 5 corresponding to the recording finish of the performance data. The 
manipulation display 5 transmits a signal corresponding to the depressed key pad to the 
controller 6. The CPU 62 receives the signal representing the recording finish of the 
performance data from the manipulation display 5 and transmits the playing stop 
instruction of the music CD to the music CD drive 1 . In response to the playing stop 
instruction, the music CD drive 1 stops playing back the music CD- A. 

[ 0073 ] 

Subsequently, the CPU 62 reads out the system exclusive event representing the 
management event and the MIDI events stored in the RAM 64 together with the 
corresponding delta time, and combines those sets of data in order to create the track chunk 
of the SMF. Moreover, the CPU 62 attaches a header chunk to the created track chunk in 
order to create the SMF. Fig. 8 is a view to show the overview of the SMF created by the 
CPU 62. The CPU 62 finishes creating the SMF and transmits the created SMF to the FD 
drive 2 together with a writing instruction. The FD drive 2 receives the write instruction 
and the SMF from the CPU 62 and writes the SMF in the loaded FD. 

[ 0074 ] 
[ 1.2.2 ] Playback operation 

Subsequently, the operations to play back the SMF recorded by the above 
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mentioned method and to synchronize the audio data of the music CD with the MIDI data 
of the SMF will be explained. The music CD used during the playback operation below 
includes the musical tune same as that of the music CD-A used in the above mentioned 
recording operation, however, version is different and the performance speeds of the 
musical tune included in the music CDs is different a little bit. Since audio effects on the 
master data of the audio data of the music CD are edited, the contents are slightly different 
from the same musical tune data in the music CD-A and the level of the audio waveform 
represented by the audio data are different. Accordingly, the music CD used in the 
playback operation, which will be explained hereunder, is called as a music CD-B in order 
to discriminate it from the music CD-A. 
[ 0075 ] 

[ 1.2.2. 1 ] Playback start manipulation 

The user loads a music CD-B on the music CD drive 1 and an FD, on which the 
SMF is recorded, on the FD drive 2. Subsequently, the user depresses the key pad of the 
manipulation display 5 corresponding to the playback start of the performance data. The 
manipulation display 5 outputs a signal corresponding to the depressed key pad to the 
controller 6. 

[ 0076 ] 

The CPU 62 receives a signal instructing the playback start of the performance 
da ta from the manipulation display 5 and transmits a transmission instruction of the SMF 
to the FD drive 2. The FD drive 2 reads out the SMF from the FD in response to the 
transmission instruction of the SMF and transmits the readout SMF to the controller 6. 
The CPU 62 receives the SMF from the FD drive 2 and stores the received SMF in the 
RAM 64. 
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[ 0077 ] 

Subsequently, the CPU 62 transmits a playback instruction of the music CD to the 
music CD drive 1 . The music CD drive 1 sequentially transmits the audio data stored in 
the music CD-B to the controller 6 in response to the playback instruction. The controller 
6 receives a pair of data in the left and right channels from the music CD drive 1 for every 
1/44100 second. Herein, the data values received from the music CD drive 1 by the CPU 
62 are represented by (r(n), l(n)). The ranges of the values r(n) and l(n) and the 
definitions of n and "sample values" used hereunder are similar to those of R(n) and L(n). 

[ 0078 ] 

[ 1.2.2.2 ] Transmission of audio data to tone generator 

The CPU 62 receives the sample values, namely, (r(0), 1(0)), (r(l), 1(1)), (r(2), 
1(2)), from the music CD drive 1 and transmits the received the sample values to the tone 
generating portion 4. The tone generating portion 4 receives the sample values from the 
controller 6 and converts it to sounds to be output. As a result, the user can listen to a 
musical tune recorded on the music CD-B. 

[ 0079 ] 

[ 1.2.2.3 ] Generation of management event 

The CPU 62 transmits the received sample values to the tone generating portion 4 
and sequentially stores the received sample values in the RAM 64. The CPU 62 stores a 
predetermined number of sample values in the RAM 64 and transmits an execution 
instruction of the management event generation process. The DSP 63 receives the 
execution instruction and executes the management event generation process, which was 
explained with reference to Fig. 4, on a series of sample values of the music CD-B 
sequentially stored in the RAM 64. As a result, the DSP 63 sequentially transmits the 
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management event to the CPU 62 at a timing when the judgment result in the step S10 in 
Fig. 4. 

[ 0080 ] 

[ 1.2.2.4 ] Playback of MIDI event 

On the other hand, the CPU 62 receives first sample values, namely, (r(0), 1(0)) 
from the music CD drive 1 and starts time measurement by setting that timing as 0 second. 
The lapse of time by the time measurement is called as "time T". At the same time, the 
CPU 62 reads out the SMF from the RAM 64 and sequentially compares the time T at that 
time with the delta time included in the SMF. When the time T coincides with the delta 
time and the event corresponding to the delta time is an MIDI event, the CPU 62 transmits 
the MIDI event to the automatic player piano 3. 

[0081 ] 

In the automatic player piano 3, the MIDI event control circuit 34 receives the 
MIDI events from the CPU 62 and transmits the received MIDI events to the tone 
generator 35 or the driving part 36. When the MIDI events are transmitted to the tone 
generator 35, the tone generator 35 sequentially transmits audio data indicative of tones of 
a musical instrument based on the received MIDI events to the tone generating portion 4. 
The tone generating portion 4 outputs the sounds of the musical tune in the music CD-B 
which is played back already and the performance by the musical tone received from the 
tone generator 35 from the speaker 44. On the other hand, when the MIDI events are 
transmitted to the driving portion 36, the driving portion 36 drives the keys and pedals of 
the piano 3 1 based on the received MIDI events. In either case, the user simultaneously 
listens to the musical tune recorded on the music CD-B and the performance with the 
musical tone by the performance information recorded in the SMF. 
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[ 0082 ] 

However, since the musical tune recorded in the music CD-B was recorded at a 
slightly different speed of the musical tune recorded in the music CD-A as mentioned 
already, the musical tune is offset from the performance by the MIDI events. Fig, 9 is a 
view to show relationship between the MIDI events and management events stored in the 
SMF and the management events generated upon playing back the music CD-B and the 
MIDI events after the timing adjustment process with respect to time. Since the clock 
frequency used for storing the audio data in the music CD-B is a relatively higher than the 
clock frequency used for storing the audio data in the music CD-A in the example in the 
drawing, the musical tune stored in the music CD-B is played back faster than the musical 
tune stored in the music CD-A upon playing back by the same music CD drive 1. 
Accordingly, when the MIDI events are transmitted to the automatic player piano 3 based 
on the delta time stored in the SMF, the automatic performance by the automatic player 
piano 3 is delayed from the musical tune in the music CD-B. The CPU 62 executes the 
timing adjustment process explained below on the MIDI events stored in the SMF in order 
to adjust the time shift between the performance by the MIDI events and the playback of 
the music CD. 

[ 0083 ] 

The timing adjusting process is a process which adjusts the delta time of the MIDI 
event based on the generation timing of the management event generated based on the 
audio data of the music CD-B and the delta time of the management event stored in the 
SMF. Hereunder, the management event stored in the SMF is called as a "management 
event A" and the management event of generated based on the audio data of the music CD- 
B is called as a "management event B". 
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[0084] 

The CPU 62 receives the management event B from the DSP 63 and stores the 
received timing in the RAM 64. On the other hand, when the time period T and the delta 
time coincides each other and the event corresponding to the delta time is a management 
event, the CPU 62 monitors whether the management event B is received during a certain 
period of time before and after the timing represented by the delta time or not. In other 
words, the CPU 62 confirms whether there is a record of receiving the management event 
B during 0.2 second before the timing represented by the delta time of the management 
event A or not. Subsequently, the CPU 62 monitors whether the management event B is 
received before lapsing 0.2 second after the timing represented by the delta time of the 
management event A or not When the management event B is received around the 
timing represented by the delta time of the management event A, the CPU judges that the 
management event A and the management event B are management events generated from 
corresponding portions of the audio data of the music CD-A and the music CD-B. Then, 
the CPU 62 multiplies (received timing of the management event B / delta time of the 
management event A) to a set of delta time of the unexecuted MIDI event and the 
management event. This is a process which adjusts the delta time in the SMF based on 
the receiving timing of the management event B. The CPU 62 repeats the adjusting 
process of the delta time by the end of the musical tune. 

[ 0085 ] 

Fig. 10 is an example of data showing the status of the timing adjusting process. 
According to the data example, the delta time of the management event A coincides with 
the time T at 0. 1 3 second. Accordingly, the CPU 62 confirms whether the management 
event B is received during a period of 0.2 second before that timing or not Since the first 
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management event B is received at 0. 11 second in this case, the CPU 62 judges that these 
events correspond each other. 
[ 0086 ] 

Subsequently, the CPU 62 multiplies (0.1 1 / 0.13) to a set of the delta time of the 
unprocessed MIDI event and the management event A by using 0. 13 second of the delta 
time of the management event A and the 0. 1 1 second of the receiving timing of the 
management event B. As a result, the delta time after the first adjustment are as follows: 
a first MIDI event: 0.41 second, a second management event: 0.69 second, a second MIDI 
event: 1.04 seconds... 

Thereafter, the delta time of the first MIDI event coincides with the time T at 0.41 
second and the CPU 62 transmits the first MIDI event to the automatic player piano 3. 

[ 0087 ] 

Since the delta time of the second management event A coincides with the time T 
at 0.69 second thereafter, the CPU 62 confirms whether the management event B is 
received during 0.£ second period before such timing or not. In this case, as the second 
management event B is received at 0.75 second, the CPU 62 judges that these management 
events correspond each other. As a result, the CPU 62 multiplies (0.75 / 0.69) to the sets 
of delta time of th? unprocessed MIDI event and the management event A as similar to the 
first management ^vent A. As a result, the delta time after the second adjustment are as 
follows: a second MIDI event: 1.14 second, a third management event: 1.34 seconds, a 
third MIDI event: £.02 seconds... 

Since the delta time of the second MIDI event coincides with the time T at 1 . 1 4 
seconds after that, the CPU 62 transmits the second MIDI event to the automatic player 
piano 3. 
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[ 0088 ] 

Hereunder, the similar processes are repeated until the end of the musical tune, a 
fourth management event A is ignored in the data example in Fig. 10 because there is not 
any management event B in time periods of 0.2 second before and 0.2 second after the 
timing represented by the delta time. As similarly, a eighth management event B is 
ignored because there is not any management event A in time periods of 0.2 second before 
and 0.2 after the generation timing. 

[ 0089 ] 

As described in the above, the MIDI event is transmitted to the automatic player 
piano 3 based on the delta time which the timing adjusting process is underwent, and as a 
result, the MEDI events are played back at approximately correct timings with respect to 
the playback of the music CD-B as shown in a lower row in Fig. 9. 

[ 0090 ] 

In contrast to the above mentioned example, when the music tune stored in the 
music CD-B is played later than the music tune stored in the music CD-A, the playback 
timings of the MIDI events are adjusted by the similar process. Accordingly, the 
explanation will be omitted. 

[0091 ] 
[ 2 ] Second embodiment 

In a second embodiment of the present invention, time codes stored in the music 
CD is used for adjusting the audio data stored in the music CD and the MIDI event stored 
in the SMF for synchronized playback. Upon the timing adjusting process, the timing 
adjustment process is executed with a calculated value by using a regression analysis after 
finishing the management event generation process of the audio data of the music CD 
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toward the end thereof. 

[ 0092 ] 
[2.1 ] Music CD drive 

Since the whole configuration, the functions of respective elements and the data 
format of the MIDI data of the second embodiment are similar to those in the first 
embodiment except a function of the music CD drive 1, the function of the music CD drive 
1 is explained only and other explanation will be omitted. 

In the second embodiment, the music CD drive 1 transmits the time codes to the 
controller 6 together with the audio data recorded in the music CD. Other features are the 
same as those of the music CD drive 1 in the first embodiment. 

[ 0093 ] 
[ 2. 2 ] Operation 

The operation of the synchronized recorder and player SS in the second 
embodiment differs from the first embodiment in following three points. 

(1) In the recording operation of the performance data, the time codes corresponding to the 
receiving timing of the events are stored as the set of delta time of the management event 
and the MIDI event stored in the SMF. 

(2) First, the audio data of the music CD is played back to the end during the playback of 
the performance data and the management event generation process is executed on the 
audio data. A regression line is calculated for the management event as a result of this 
and the time information stored in the SMF; the delta time corresponding to the delta time 
is adjusted based on the regression line. After finishing the timing adjusting process, the 
music CD is played back and the MIDI events are synchronously played back for the 
playback thereof. 
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(3) The MIDI events are transmitted to the automatic player piano 3 based on the time 
clock transmitted from the music CD drive 1 during the playback of the performance data. 
[ 0094 ] 

Other operations in the second embodiment are similar to those of the first 
embodiment and the detailed explanation will be omitted. It is assumed that the music 
CD-A and the music CD-B are used for a recording operation and a playback operation, 
respectively, as similar to the first embodiment in the following explanation. The format 
of the time code is represented by hour, minute, second and frame and the time information 
represented by the time code is represented in second as similar to the delta time recorded 
in the SMF in the following explanation for the sake of the simplicity. 

[ 0095 ] 
[ 2.2.1 ] Recording operation 

In the synchronized recorder and player SS in the second embodiment, when the 
user instructs the recording start of the performance data through the manipulation display 
5, the audio data in the music CD-A is sequentially transmitted to the controller 6 from the 
music CD drive 1 together with the time codes. 

[ 0096 ] 

In the controller 6, the CPU 62 sequentially transmits the received audio data to 
the tone generating portion 4 and the musical tune of the music CD-A is output as sounds 
from the tone generating portion 4. On the other hand, the CPU 62 sequentially stores the 
received sample values in the RAM 64 together with the time codes. 

[ 0097 ] 

The CPU 62 stores the predetermined number of sample values in the RAM 64 
and transmits an execution instruction of the management event generation process to the 
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DSP 63. The DSP 63 receives the execution instruction and executes the management 
event generation process on a series of sample values of the music CD-A which are 
sequentially stored in the RAM 64. The management event generation process in the 
second embodiment is the same as the management event generation process in the first 
embodiment explained with reference to Fig. 4. As a result, the DSP 63 sequentially 
transmits the management event to the CPU 62 at a timing when the judgment result in the 
step S 1 0 in Fig. 4 becomes No. The CPU 62 receives the management event and 
generates the system exclusive event representing the management event. Subsequently, 
the CPU 62 converts the time represented by the time code received from the music CD 
drive 1 in the format of the delta time immediately before receiving the management event 
and after that, correlates the delta time to the system exclusive event represented by the 
management event to be stored in the RAM 64. 
[ 0098 ] 

On the other hand, the user listens to the musical tune of the music CD-A output 
as sounds from the tone generating portion 4 and plays on the piano 3 1 . The information 
by the user performance is transmitted to the CPU 62 through the MIDI event control 
circuit 34 as the MIDI events. The CPU 62 receives the MIDI events from the MIDI 
event control circuit 34, converts the time represented by the time code received from the 
music CD drive 1 immediately before, and correlates the delta time to the MIDI event to be 
stored in the RAM 64. 

[ 0099 ] 

Thereafter, when the user instructs the recording end of the performance data, the 
music CD drive 1 stops playing the music CD-A. The CPU 62 reads out the system 
exclusive event representing the management event stored in the RAM 64 and the MIDI 
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event together with the corresponding to the delta time and combines the data in order to 
generate the SMF similar to those shown in Fig. 9. The delta time of the SMF in the 
second embodiment is based on the time code of the music CD-A and is not based on the 
measured time of the CPU 62. The SMF generated in such a way is transmitted to the FD 
drive 2 and is written in the FD loaded in the FD drive 2. 

[0100] 
[ 2.2.2 ] Playback operation 

Subsequently, operations for playing back the SMF, which is recorded by the 
above mentioned method, and for synchronizing the audio data of the music CD-B and the 
MIDI data of the SMF will be explained. 

The user instructs the playback start of the performance data by using the 
manipulation display 5; the SMF recorded in the FD is transmitted from the FD drive 2 to 
the CPU 62 and the CPU 62 stores the received SMF in the RAM 64. Subsequently, the 
music CD drive 1 starts playback of the music CD-B and the audio data and the time code 
recorded in the music CD-B are sequentially transmitted to the controller 6. The CPU 62 
stores the audio data in the RAM 64 together with the time code. However, the CPU 62 
does not transmit the received audio data to the tone generating portion 4 and, accordingly, 
the user does not hear the musical tune of the music CD-B. 

[0101 ] 

The CPU 62 stores the predetermined number of sample values in the RAM 64 
and transmits an execution instruction of the management event generating process to the 
DSP 63. The DSP 63 receives the execution instruction and executes the management 
event generation process on the series of sample values of the music CD-B which are 
sequentially stored in the RAM 64. As a result, the DSP 63 sequentially transmits the 
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management event to the CPU 62 at a timing when the judgment result in the step S10 in 
Fig. 4 becomes No. The CPU 62 receives the management event, converts the time 
represented by the time code received from the music CD drive 1 immediately before in 
the format of the delta time and after that, and stores the delta time in the RAM 64. 
Hereunder, the management event stored in the SMF is called as a "management event A"; 
the delta time stored in the SMF is called as a "delta time A"; the management event 
generated with the audio data of the music CD-B is called as a "management event B"; and 
the delta time stored upon receiving the management event is called as a "delta time B". 
[0102] 

When the playback of the music CD-B comes to an end, the music CD drive 1 
finishes the playback of the music CD-B. Subsequently, the CPU 62 reads out the delta 
time A and the delta time B from the RAM 64 and executes the adjusting process of the 
delta time as similar to the timing adjusting process in the first embodiment. Fig. 1 1 
shows data examples of the delta time adjustment in the second embodiment. In Fig. 1 1 , 
the receiving timings of the delta time and the management event B are based on the time 
codes of the music CD-B and it is different from those in the first embodiment. The 
adjustment of the delta time is not executed on the MIDI events. Other points are similar 
to the those in the timing adjustment process in the first embodiment and the explanation 
will be omitted. The delta time A and the delta time B are correlated each other by the 
adjustment process of the delta time. A value of the delta time A and a value of the delta 
time B in a nth pair are represented as A(n) and B(n), respectively. The pair is 
represented as (A(n), B(n)). Accordingly, these are represented as (A(l), B(l) = (0. 13, 
0.11), (A(2), B(2) = (0.81, 0.75), (A(3), B(3) = (1.45, 1.36), ... 

[0103] 
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Subsequently, the CPU 62 calculates a regression line for the (A(n), B(n)) by the 
least-square method. A regression curve and other approximated curves may be 
calculated instead of the regression line. Actually, the regression line is calculated for 
(A(n), B(n)) acquired for the audio data of the whole musical tune and the it is assumed 
that nine pairs of data acquired from Fig. 1 1 are used in order to calculate the regression 
line. 

[0104] 

Fig. 12 is a graph showing nine pairs of (A(n), B(n)) and a regression line therefor. 
A formula representing the regression line is B = 0.94 14A - 0.006. This formula indicates 
that the starting timing of the performance data of the SMF is delayed from the starting 
timing of the audio data of the music CD-B with 0.006 second and the performance speed 
is slower than that with 5.86 % after setting the time code of the music CD-B as a base. 
Accordingly, the CPU 62 adjusts the timing with respect to the delta time by substituting 
the value of the delta time of the MIDI event recorded in the SMF for A in the formula of 
the calculated regression line. The delta time before the timing adjustment and the delta 
time after the adjustment are shown in Fig. 13. The timing adjustment by the CPU 62 
decreases the delta time of each event with 5.86 % and moves them forward with 0.006 
second. The CPU 62 renews the delta time of the MIDI events of the SMF with the delta 
time after finishing the timing adjustment. 

[0105] 

The CPU 62 finishes the timing adjustment process and transmits a playback 
instruction of the music CD-B to the music CD drive 1 again. The CD drive 1 starts 
playback of the music CD-B in response to the playback instruction and the audio data and 
the time code stored in the music CD-B are sequentially transmitted to the controller 6. 
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The CPU 62 sequentially transmits the received audio data to the tone generating portion 4 
and the tone generating portion 4 outputs the musical tune of the music CD-B as sounds. 
[0106] 

The CPU 62 sequentially transmits the received audio data to the tone generating 
portion 4 and at the same time, compares the time information represented by the delta 
time of the MIDI events with the time information represented by the time code of the 
music CD-B transmitted from the music CD drive 1, and when they are matched, transmits 
the MIDI event corresponding to the matched delta time to the automatic player piano 3. 
The automatic player piano 3 receives the MIDI events and executes the automatic 
performance associated with the MIDI events by transmitting the audio data of 
electronically synthesized musical tones from the tone generator 35 to the tone generating 
portion 4 or by driving the keys and pedals of the piano 3 1 by the driving portion 36. As 
a result, the user can listen to the musical tune recorded in the music CD-B and the 
performance by the performance information recorded in the SMF simultaneously. 

[0107] 
[ 3 ] Modifications 

The above mentioned first embodiment, second embodiment and third 
embodiment are mere illustrations of the embodiments of the present invention, and 
various modifications are available without departing from the feature of the present 
invention. Modifications will be shown hereunder. 

[0108] 
[3.1] First modification 

In the first modification, elements of the synchronized recorder and player SS 
are not located in the same device and are separated into groups to be located. 
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For example, they are separable into following respective groups: 

(1) music CD drive 1 

(2) FD drive 2 

(3) automatic player piano 3 

(4) mixer 4 1 and D/A converter 42 

(5) amplifier 43 

(6) speaker 44 

(7) manipulation display 5 and controller 6 

Further, the controller 6 may be separated into a device for recording operations 
only and a device for playback operations only. 
[0109] 

The element groups are connected with audio cables, MIDI cables, optical audio 
cables, USB (Universal Serial Bus) cables and dedicated control cables. The FD drive 2, 
the amplifier 43 and speakers 44 may be commercially available ones. 

According to the first embodiment, the location flexibility of the synchronized 
recorder and player SS is enhanced and the user does not need to prepare the whole new 
components of the synchronized recorder and player SS to reduce the cost by preparing the 
necessary components only. 

[0110] 
[ 3.2 ] Second modification 

In a second modification, the synchronized recorder and player SS does not 
include the music CD drive 1 and the FD drive 2. On the other hand, a communication 
interface has a function connectable to the LAN (Local Area Network) and is connected to 
external communication devices through the LAN and WAN. The controller 6 has an HD 
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(Hard Disk). 

[0111] 

The controller 6 receives the digital audio data including the audio data and the 
time codes from other communication devices through the LAN and records the received 
audio data in the HD. As similarly, the controller 6 receives the SMF created in 
association with the audio data from other communication devices through the LAN and 
records the received SMF in the HD. 

[0112] 

The controller 6 reads out the digital audio data from the HD instead of receiving 
the audio data and the time codes of the music CD from the music CD drive 1 . The 
controller 6 executes the similar operations on the HD instead of writing and reading out 
the SMF into or from the FD drive 2. 

According to the second modification, the user is capable of transmitting and 
receiving the digital audio data and the SMF through the LAN to the communication 
device which is geographically remote therefrom. The LAN may be connected to the 
wide area Communication network such as the Internet. 

[0113] 

[ EFFECTS OF THE INVENTION ] 

As explained in the above, according to the present invention, the synchronized 
play back of the performance data is allowed at a correct timing for the audio data of 
difference versions in which the audio data having different starting points of the same 
musical tune. Accordingly, a different set of performance data is not necessary for a 
different version of the same musical tune and the data generation and data management 
are simplified. 
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[0114] 

There may be a case that recording levels are different in the different versions of 
the same musical tune, the management event used for adjusting the playback timing of the 
performance data in the present invention is generated at the same timing irrelevant to the 
recording level of the musical tune. This is because the two indices used for generating 
the management event vary in proportion to the recording levels so that the relation with 
respect to the magnification is unchanged. Accordingly, the audio data of the version 
having different recording level is synchronously played back with the performance data at 
correct timings. 

[ BRIEF DESCRIPTION ON THE DRAWINGS ] 

[ Fig. 1 ] The view to show the configuration of the synchronized recorder 

and player SS implemented by the first embodiment and the second embodiment of the 
present invention. 

[ Fig. 2 ] The view to show the format of the MIDI event. 

[ Fig. 3 ] The view to show the structure of SMF. 

[ Fig. 4 ] The flowchart of the management event generation process 

implemented by the first embodiment and the second embodiment of the present invention. 

[ Fig. 5 ] The view to show the configuration of the comb filter 

implemented by the first embodiment and the second embodiment of the present invention. 

[ Fig. 6 ] The view to show the relationship between the middle term 

index, the long term index and the management event with respect to time implemented by 
the first embodiment and the second embodiment of the present invention. 

[ Fig. 7 ] The view to show the relationship between the generated 

management events and the generated MIDI events with respect to time implemented by 
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the first embodiment and the second embodiment of the present invention. 

[ Fig. 8 ] The view to show the overview of the SMF implemented by the 

first embodiment and the second embodiment of the present invention. 

[ Fig. 9 ] The view to show the relationship between the MIDI events 

before the timing adjustment, the MIDI events after the timing adjustment, and the 
management event with respect to time implemented by the first embodiment of the 
present invention. 

[ Fig. 10 ] The view to show the example of data showing states of the 

timing adjustment process implemented by the first embodiment of the present invention. 

[ Fig. 1 1 ] The view to show the example of data showing states of the 

timing adjustment process implemented by the second embodiment of the present 
invention. 

[ Fig. 1 2 ] The view to show the regression line for the delta time of the 

management event implemented by the second embodiment of the present invention. 

[ Fig. 13 ] The view to show the example of data showing states of the 

adjustment process on the delta time of the management event implemented by the second 
embodiment of the present invention. 

[ Fig. 14 ] The view to show the relationship between the MIDI events 

before the timing adjustment, the MIDI events after the timing adjustment, and the 
management event with respect to time implemented by the second embodiment of the 
present invention. 

[ EXPLANATION ON REFERENCES ] 
1 ... music CD drive, 2 ... FD drive, 3 ... automatic player piano, 4 ... tone 
generating portion, 5 ... manipulation display, 6 ... controller, 31 .... piano, 32 ... key 
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sensor, 33 ... pedal sensor, 34 ... MIDI event control circuit, 35 ... tone generator, 
36 ... driving portion, 41 ... mixer, 42 ... D/A converter, 43 ... amplifier, 44 ... 
speaker, 61... ROM, 62 . . . CPU, 63 ... DSP, 64 . . . RAM, 65 . . . communication 
interface 
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[ DOCUMENT NAME ] ABSTRACT DOCUMENT 
[ ABSTRACT ] 

[ PROBLEM ] To provide a recorder, a player, a recording method, a playback method and 
program which permit synchronized playback of the performance data at correct timings 
for plural sets of audio data of the same musical tune having different speeds. 
[ SOLVING MEANS ] A controller 6 records MTDT data of performance through a piano 
3 1 , which is played with the playback of a music CD, in an SMF. At this time the 
controller 6 compares values represented by components equal to or lower than 100 Hz 
with those equal to or lower than 1 Hz of the audio data of the music CD and stores 
management events generated as a consequence in the SMF. Subsequently, the controller 
6 plays back the MIDI data stored in the SMF together with the playback of the music CD. 
At this time, the controller 6 generates a management event from the audio data of the 
music CD, and compares the timings of the management events stored in the SMF and the 
timings of the generated management events in order to adjust the playback timings of the 
MIDI data. 

[ SELECTED FIGURE ] Fig. 1 



